Skip to content

Piyush Routray

  • Twitter
  • LinkedIn
  • About Piyush

Weekdays cron job, but for a particular date range.

Cron is a very simple, yet useful job scheduler on linux. To schedule a job for weekdays, one can easily…

cron, linux

Managing Databricks Secrets using Windows.

Databricks has a method of Secret Management which allows users to save sensitive data such as JDBC connection or Snowflake…

Databricks, DevOps, Windows

PySpark DataFrame ordering based on multiple columns

How to deal with the requirement to order a DataFrame using more than one column simultaneously? Also, consider some values…

Data Engineering, Databricks, PySpark

How to retain the first row of each ‘group’ in a PySpark DataFrame?

For a given dataframe, with multiple occurrence of a particular column value, one may desire to retain only one (or…

Databricks, PySpark

Installing ‘s3cmd’ on centOS 7+

AWS S3 is one of the oldest offerings for ‘object storage’, from Amazon AWS and when working through a terminal,…

AWS S3, centOS 7, s3cmd

How to split contents of a column into *only two* by a delimiter?

While working with PySpark, I came across a requirement, where data in a column had to be split using delimiters…

PySpark

Change java for Cloudera cluster using ambari

If you are using Hortonworks (presently Cloudera) cluster, the default value for java is Oracle JDK 1.8. If you have…

ambari, cloudera, Hortonworks, java, OpenJDK, OracleJDK

How to encrypt files before sharing online?

As a Data Engineer, one faces the need to share files securely over the internet. An easy way of doing…

Data Engineering, PGP, security

Posts navigation

Older posts

Browse by Category

ambari AWS Firehose AWS S3 centOS 7 cloudera cron Databricks Data Engineering datanode DevOps docker hadoop HDP hive Hortonworks java jenkins kubernetes linux OpenJDK OracleJDK PGP PySpark Python s3cmd security Snowflake stackoverflow Windows

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 433 other subscribers
My Tweets
Create a website or blog at WordPress.com
  • Follow Following
    • Piyush Routray
    • Already have a WordPress.com account? Log in now.
    • Piyush Routray
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar
 

Loading Comments...