Intermediate LevelCloud Computing (AWS/GCP/Azure)
What is AWS Glue Data Catalog and how do Glue Crawlers discover schema metadata?
❓ Interview Prompt: Explain AWS Glue serverless ETL, Glue Crawlers, Glue Data Catalog metadata tables, and PySpark execution jobs.
AWS Glue is a fully-managed serverless data integration (ETL) service that simplifies data preparation for analytics.
Key Components:
1
AWS Glue Data Catalog: A centralized metadata repository storing table definitions, schema formats, and partition locations across S3 data lakes.
2
Glue Crawlers: Automated processes that inspect S3 objects, infer data schemas (Parquet, JSON, CSV), and populate/update Data Catalog tables automatically.
3
Glue Spark Jobs: Serverless PySpark or Scala ETL jobs that transform, clean, and enrich multi-terabyte datasets.
Code Implementation Example
python
# PySpark AWS Glue ETL Job snippet converting CSV to Parquet
from awsglue.context import GlueContext
from pyspark.context import SparkContext
sc = SparkContext()
glueContext = GlueContext(sc)
# Load DynamicFrame from Glue Data Catalog
datasource = glueContext.create_dynamic_frame.from_catalog(
database="analytics_db",
table_name="raw_csv_logs"
)
# Write converted Parquet format back to S3
glueContext.write_dynamic_frame.from_options(
frame=datasource,
connection_type="s3",
connection_options={"path": "s3://company-data-lake/parquet_logs/"},
format="parquet"
)💡PySpark AWS Glue ETL job converting S3 CSV data to Parquet format.
🎯 Interview Key Takeaway
AWS Glue Crawlers infer S3 data schemas to populate the central Glue Data Catalog for Athena and Spark analytics.
#AWS Glue#ETL#Data Catalog#Data Lake#Intermediate
Back to all Cloud Computing (AWS/GCP/Azure) Questions→