Daniel B

data-engineering-practice. Data Engineering Practice Problems

dataEngineeringTemplate. Template for Data Engineering and Data Pipeline projects

datahobbit. A Rust based data/CSV/Parquet file generator

drainage. Rust + Python Lake House Health Analyzer | Detect • Diagnose • Optimize • Flow

tinytimmy. A simple and easy to use Data Quality (DQ) tool built with Python.

sniffer. csv and flat-file sniffer built in Rust.

DataEngineeringProjects. Some example projects for Data Engineers to build, end-to-end.

DuckDBwithAWSLambda. Using DuckDB with AWS Lambda to process Delta Lake data

unitTestPySpark. how to unit test your PySpark code

reepicheep. This is a `Rust` based package to help with the management of complex medicine (pill) management cycles.

lakescum. A Python package to help Databricks Unity Catalog users to read and query Delta Lake tables with Polars, DuckDb, or PyArrow.

GreatExpectationsWithDatabricks. Getting Great Expectations setup to run on DataBricks with Spark Dataframes.

PolarsVsPySpark. can Polars crunch 27GBs of data faster than Pyspark?

RustForDataPipelines. Testing out if Rust can be used for a normal Data Engineering Pipeline.

PythonVsRustAWSLambda. Testing the runtime difference between Python and Rust for AWS Lambda.

polars-DeltaLake. Trying out the Dataframe Polars library with Delta Lake ... feat Python.

agenticAIagent. An example of building an Agentic AI agent system.

RustOnApacheAirflow. The ultimate Data Engineering Chadstack. Apache Airflow running Rust. Bring it.

learnDataEngineering. Sample Project to Learn Data Engineering

polarsVpandasOnAwsLambda. Using Polars and Pandas on AWS Lambda to process data.

DuckdbAndDeltaLake. Learning how to query remote s3 Delta Lake with DuckDB.

tryingoutDuckLake. trying out DuckLake from DuckDB with Postgres

DataFrameShowDown. Polars vs Spark vs Pandas vs DataFusion. Guess who wins?

fine-tune-openLLaMA. This repo shows how to fine tune openLLaMA (7b) model on a GPU.

DuckDBwithJSONfiles. processing JSON files in S3 with DuckDB

PandasVsPolars. Try some common functions between Pandas and Polars.

SysproInvoicing. use Python to Invoice in Syspro ERP System

pyarrow-v-duckdb-v-polars. Compare pyarrow to duckdb to polars for writing data pipelines.

rustAsyncExample. A quick example of using Rust to do async HTTP requests/downloads.

AirflowVsDagster. Comparing Apache Airflow to Dagster

GreatExpectationsWithSpark. Learning to setup a Great Expectations project using Apache Spark

MotherDuckwithApacheAirflow. Combine MotherDuck + DuckDB with Apache Airflow

puddleglum. Rust based package for answer questions about s3 buckets and files

dqxDatabricksDataQuality. trying out the new dqx Data Quality library from Databricks Labs

gRPCwithPython. Introduction to gRPC with Python.

TheBearVsTheDuck. Compare DuckDB v Polars for Data Pipelines.

datafusion-sql-cli. Playing around and making ETL tools with Datafusion's CLI SQL tool.

RustDataFusion. Trying out Rust's DataFusion, compare to Apache Spark.

DataEngineeringWithFortran. Trying to use Fortran to write a data pipeline

sparklepop. SparklePop is a simple Python package designed to check the free disk space of an AWS RDS instance.

pythonRustLambda. using Delta Lake Python bindings for delta-rs to write large CSV files to Delta Lakes.

PolarsDateTimeManipulation. Polars date and time manipulation

try-smallpond. trying out smallpond that's built on duckdb

GolangDataFrames. playing with DataFrames in Golang, compare it to Python.

graphRS. Building a Network/Graph from scratch, and understanding it with Rust.

kafkaClusterWithPython. create a 3 node Kafka cluster, interact with Python client.

sparkShufflePerformance. testing the performance of Spark shuffle configurations

pySparkSQLContext. Learning to use SQLContext with PySpark.

DuckDBvsPolars. Comparing the performance of DuckDB to Polars

PrefectIntroduction. Trying out Prefect as compared to Airflow.

airflow-kubernetes. Running Airflow inside Kubernetes

SparkHadoopCluster. create your own Apache Spark cluster with Hadoop/HDFS installed.

sparkMachineLearningExample. An example of a Spark Machine Learning Pipeline in PySpark.

GolangVsRust. Writing Word Counter with both Golang and Rust

RayonWithRustVsPython. Trying on Rayon with Rust vs Python Thread and ProcessPools.

postgresInsertPerformance. Testing Postgres Insert Performance

testApacheArrow. Trying out Apache Arrow, compare to Polars.

s3cloudStorage_Golang_Python_Rust. Golang, Rust, and Python working with s3 files.

DSAforTheRestOfUs. Introduction to DSA (Data Structures and Algorithms) with Rust.