Databricks Connect

Databricks Connect allows you to connect popular IDEs such as Visual Studio Code, PyCharm, IntelliJ IDEA, notebook servers, and other custom applications to Databricks clusters.

Databricks Connect is a client library for the Databricks Runtime. It allows you to write code using Spark APIs and run them remotely on a Databricks cluster instead of in the local Spark session.

Refer to the official docs at https://docs.databricks.com/en/dev-tools/databricks-connect/python/index.html