Weekdays cron job, but for a particular date range.

Cron is a very simple, yet useful job scheduler on linux. To schedule a job for weekdays, one can easily…

Managing Databricks Secrets using Windows.

Databricks has a method of Secret Management which allows users to save sensitive data such as JDBC connection or Snowflake…

How to deal with the requirement to order a DataFrame using more than one column simultaneously? Also, consider some values…

For a given dataframe, with multiple occurrence of a particular column value, one may desire to retain only one (or…

AWS S3 is one of the oldest offerings for ‘object storage’, from Amazon AWS and when working through a terminal,…

While working with PySpark, I came across a requirement, where data in a column had to be split using delimiters…

If you are using Hortonworks (presently Cloudera) cluster, the default value for java is Oracle JDK 1.8. If you have…

As a Data Engineer, one faces the need to share files securely over the internet. An easy way of doing…