Techniques to create trust in your data solution

Photo by Zan on Unsplash

A core responsibility of a data engineer is to build data pipelines. The flow of data from one location to another with mutations along the way is a primary focus for any data engineer. However, when you change data, you make people nervous. Even though you are actively changing the data from its raw form into something usable, it doesn’t change the fact that it is no longer the data that was collected. In order to mitigate these fears, it is vital that data engineers build as many data integrity measures into the solution as possible.

Techniques in quality assurance

A byte-by-byte comparison is…

My transition between two seemingly unrelated fields

Photo by Ian Schneider on Unsplash

The unique field of data engineering

Data engineering is unique in that it does not have a formal education requirement. You can have a PhD in physics or no higher education at all. It is the skills and experience that define your ability to succeed as a data engineer. I, myself, have a degree in chemical engineering. I had no exposure to traditional data engineering prior to my first data engineering role. However, through self-study (and a bit of luck), I was able to thrive as a data engineer and I found a career that I thoroughly enjoy.

The steps I took

As I alluded to above, everybody’s journey to…

The hidden benefits of code reviews for software developers

People talking at desk
People talking at desk
Photo by Charles Deluvio on Unsplash.

Ah, the dreaded code review. A necessary evil despised by all software engineers. To a reviewer, code reviews are just one more task on a to-do list that is already too long. To a reviewee, code reviews act as a barrier preventing any further progress until their code has been reviewed. I mean, why review my code when it is already perfect… right?

When I first started my career, I admit that I had a similar attitude towards code reviews. However, as I progressed in my career, code reviews became less of a burden and more of an opportunity. …

A primer in database normalization

Photo by Glenn Carstens-Peters on Unsplash

Data modeling is an essential skill in all data engineers’ toolsets. However, early in my career, I found data modeling to be extremely difficult: partly because there is not a formulaic approach to modeling data and partly because the information available is based on set theory, which I have never had the pleasure (displeasure?) of studying. However, I found data modeling to become increasingly easier as I gained experience. While it can never be an exact science, given the business interdependencies, I found an approach that has worked for me throughout the years.

Designing a data model is not an…

The essential skills required for all data engineers

Photo by Markus Spiske on Unsplash

The explosion of data volumes and throughputs in the business climate have led companies in the direction of data-driven decision making. Historically, data scientists have reaped the benefits of this new found direction. However, as the role of a data scientist narrows, a new role is coming to the forefront of the data field.

Data engineering is a relatively new term and, as such, it has yet to standardize its definition across the various industries. Some companies will hire a data engineer with responsibilities more akin to a database administrator while other companies will expect data engineers to have data…

Three Questions to Consider Before Starting a Business

Photo by Matt Walsh on Unsplash

Starting a business is both exciting and daunting. It is the beginning of a legacy, and it requires you to harden yourself against inevitable rejection and 80 hour work weeks. It is crucial that we start a business with a proper foundation; else, we risk complete and utter failure. Consider the following three questions prior to embarking on your entrepreneurial journey:

1. How are you creating value?

The goal of any business is to provide value. You must identify an area that is suboptimal and construct your business around it via products and/or services. The more value you can provide your target audience, the higher probability…

Travis Cooper

Engineer and entrepreneur dedicated to learning something new every day.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store