BigQuery (GCP)....
in AWS we have redshift similar to bigquery in GCP
Hieriercy of BigQuer-->
GCP project --> we have a dataset (similar to database) --> contains the tables..
In realtime how it works is - if we are working in insurance company....we have different types of insurances like health insurance, home insurance, car insurance etc..so this is organized?
So lets say, we have a GCP project --> XIG insurance_dev , PROD etc..
within dev project, based on hieriercy we will create a health dataset and with in that i will have customore_UK table, customer_italy table..like this so many customers table within the health dataset.
So the same structure can be applied in PROD aswell.....
gcloud sdk - to interact from local machine...for that i need to install it..
optimize the queries in bigData....
BigQuerry - fully managed, serverless data warehouse, provided by google cloud, that allows to store, query and analyze large datasets using sql queries.
key features and when to use - serverless (infrastructure managed by google itself), fast (distributed parallel processing, built in cache to avoid recomputation, stores data in columnar format which means columns needed for queries only will be read), scalable (handles any data size efficiently, and dynamic allocation makes that possible by adjusting resources automatically), ML (using bigquery ML to train models on large datasets), Real time analytics (for fast business insights), handling big data (dataset in terabytes or petabytes), data warehouse (store and query structured data efficiently)
"Big Fast Servers Scale with ML & Real-time Data"
- Big → BigQuery (fully managed and serverless)
- Fast ⚡ → Parallel processing, caching, and columnar storage
- Serverless → No server management required, infrastructure handled by Google
- Scale 📈 → Automatically adjusts resources for any data size (dynamic allocation)
- ML 🤖 → Machine learning with BigQuery ML
- Real-time 📊 → Fast insights for business decisions
- Data 📂 → Handles big data (terabytes to petabytes) efficiently
so big query can be used to store large volume of data and also to process it quickly??
-
Storage: BigQuery is designed to store massive datasets. It can handle petabytes (and even exabytes) of data, making it suitable for large-scale data warehousing.
-
Processing: BigQuery is also excellent at processing data quickly. It uses a powerful SQL query engine that can analyze large datasets in seconds. This makes it ideal for tasks like:
- Data analysis
- Generating reports
- Transforming data
- Supporting real-time analytics
Simple Bullet Points (for Quick Reference in an Interview):
WSP (2018-2024):
- Migrated CFD simulation data to Google Cloud and used BigQuery for storing large datasets.
- Likely leveraged BigQuery's SQL for analyzing simulation data, supporting insights and decision-making with BI tools (like Tableau).
LexisNexis (2024-2025):
- Used BigQuery SQL for transforming and cleaning data.
- Integrated BigQuery into a cloud-based data pipeline for efficient data processing and orchestration.
=======================================================================
pythoN
---> modulus operator % >> n%2 !=0 using this we can check if the number is een of not..i.e the reminder is zero means even , if not odd
Integer Division:
print(a // b): Performs integer division ofabybusing the//operator. This operator returns the quotient (the whole number result) of the division.
Float Division:
print(a / b): Performs float division ofabybusing the/operator. This operator returns the quotient as a floating-point number (a number with a decimal).
