Microsoft Fabric – check if a path exists using pyspark

My wife is always quoting that old British comedy show called dinner ladies, in which a character used to say:-

there are two ways to Urmston

Dinner ladies – BBC sitcom

This occurs every time something comes up where there are 2 ways to do it or get there. So, I wonder what she’d make of this, since there are 2 ways to check if a path exists in Microsoft Fabric using pyspark. Actually, maybe there might be more, but depending on the type of path you’re checking you might find some of these methods useful.

In this example I’m making an assumption you’re using a lakehouse. Under the ‘Files’ section you can right click on any file to get the path (or load the file). This hopefully familiar box of choices pops up:-

These give you 2 different ways to get path to the same file, for example:-

• Relative path for Spark = “Files/some_folder/some_file.csv”
• File API path = “/lakehouse/default/Files/some_folder/some_file.csv”

If you need to potentially deal with someone passing either path type you could try this little function (or just hack the string value instead to remove or add the ‘/lakehouse/default/’ part then use either method!). Personally I chose the hard way, just so I could play about with different ways of doing things. Messing about is always a good way to learn new things and ways of doing stuff. I find that quite often it gets you a deeper understanding of the subject as opposed to just following the easiest path (no pun intended).

Note, I’ve ignored the ABFS path as that is a URL, we’re just interested in file paths for this example 😉

import os.path

def path_exists(path_to_check):
    # If its a relative path from the files use method 1 where 
    # we need to use the ugly 'hack' involving 'ls' since os.path.exists 
    # doesn't work with relative paths for spark, only the 'default' lakehouse paths
    if path_to_check.startswith("Files/"):
        try:
            # If we can list the path, obviously it exists. If it doesn't then this
            # throws an exception which we can catch below ;-)
            mssparkutils.fs.ls(path_to_check)
            return True
        except:
            # If the path doesn't exist, we just catch the exception thrown above
            # and now we know that the path doesn't exist
            return False
    
    # Otherwise we can just use the os.path method ;-)
    else:
        return os.path.exists(path_to_check)

# Test relative path
path_to_test = "Files/some_folder/some_file.csv" # Relative path for spark

if path_exists(path_to_test):
    print("exists")
else:
    print("doesn't exist")

# Test the other way
path_to_test = "/lakehouse/default/Files/some_folder/some_file.csv" # File API path

if path_exists(path_to_test):
    print("exists")
else:
    print("doesn't exist")

There we go, hope someone finds this useful… till next time.