Weekdays cron job, but for a particular date range.

Cron is a very simple, yet useful job scheduler on linux. To schedule a job for weekdays, one can easily … More

Managing Databricks Secrets using Windows.

Databricks has a method of Secret Management which allows users to save sensitive data such as JDBC connection or Snowflake … More

How to deal with the requirement to order a DataFrame using more than one column simultaneously? Also, consider some values … More

For a given dataframe, with multiple occurrence of a particular column value, one may desire to retain only one (or … More

AWS S3 is one of the oldest offerings for ‘object storage’, from Amazon AWS and when working through a terminal, … More

While working with PySpark, I came across a requirement, where data in a column had to be split using delimiters … More

If you are using Hortonworks (presently Cloudera) cluster, the default value for java is Oracle JDK 1.8. If you have … More

As a Data Engineer, one faces the need to share files securely over the internet. An easy way of doing … More