Best Practices Streaming Google Professional Data Engineer GCP
- Using composable transforms having business logic creates pipelines to easily maintain, troubleshoot and develop with.
- Ensure they have a clear owner
- Have a clear way to understand and document how changes to those transforms will affect all pipelines that use them.
- Always use the name field to assign a useful, at-a-glance name to the transform.
- Ensure that all transforms have versions and that all pipelines also have versions.
- Ensure you have a consistent policy to deal with errors and issues across all of transforms.
- End-to-end data pipeline testing should include lifecycle testing, particularly analyzing and testing all the different update/drain/cancel options.
- In addition to the standard metrics, build custom metrics into pipeline as per
- Create backups of pipelines
- Create multiple replica pipelines
Google Professional Data Engineer (GCP) Free Practice TestTake a Quiz