Python OS module in detail with examples & syntax

__author__ = "Narendra Boyina"

"""
The OS module allows us to interact with the underlying operating system in several different ways.
- Navigate the file system
- Get file information
- Look up and change the environment variables- Move files around
- Many moreTo begin, import the os module. This is a built in module, 
no third party modules need to be installed.
"""

import os,time
from datetime import datetime

"""dir() will display all of the methods and attributes  of os module"""
# print(dir(os))

"""To get current working directory"""
# path = os.getcwd()  # @Answer /home/narendrab/Python/My Practice# print(path)

"""To Change directory, from one path to another path. this requires a path to change to
Syntax ==> os.chdir(path)"""
# path = "/home/narendrab/pytest"# os.chdir(path)
# print(os.getcwd())  # will display  /home/narendrab/pytest


# To print list of directories & files names in provided path,
# Syntax ==> os.listdir(path) but by default path is in the current directory
# print(os.listdir("/home/narendrab/Python/"))

# To create a new directory. Syntax ==> mkdir("Directory/folder name")
# os.mkdir("Narendra")


# if you want to create multiple directories at once
# (directory inside directory inside directory ..ectc)
# Syntax ==> os.makedirs("directory_Name/Sub_directory_1/Sub_directory_2")
# os.makedirs("Narendra/Bhagyasree/Raj")



# Remove single directory/ multiple directories
# Syntax ==> os.rmdir(file)


# Dirs created ==>path changed ==>printed path after changed==> came back one stepDir
# path = "Narendra/Bhagyasree/Raj"
# os.makedirs(path)  
# creation of directories/sub-dir
# print("Directories were created just check in PWD/(Pycharm tree structure)")
## os.chdir(path)  # enter in to the directories created just now
# print(os.getcwd())  # will display PWD
## os.chdir("/home/narendrab/Python/My Practice/Narendra/Bhagyasree")
# come back without giving path
# os.rmdir("Raj")  # Removes the last directory in the path
# print(os.getcwd())

# # Syntax ==> removedirs(file)  # Removes intermediate directories if specified

# create a file with write mode
# fp=open("test.txt", "w+")
# fp.write("Hi,\n This file was created just for execution purpose")
# fp.close()


# # Rename a file or folder
# os.rename("test.txt","Narendra.txt")  # This renames text.txt to Narendra.txt
## # Look at info about files# print(os.stat("Narendra.txt"))
# # Useful stat results: st_size (bytes), st_mtime (time stamp)
# print(os.stat("Narendra.txt").st_size)  # display size of the file
# print(os.stat("Narendra.txt").st_mtime)  # display the last modification time of the file (but can't understand by user)
# last_mod_time = os.stat("Narendra.txt").st_mtime
# print(datetime.fromtimestamp(last_mod_time))  # Now user can understand recently file modified time




# # To see entire directory tree and files within
# os.walk is a generator that yields a tuple of 3 values as it walks the directory tree
# path = os.getcwd()
# # for dirpath, dirnames,filenames in os.walk(path):
#     print("current path : ",dirpath)
#     print("Directories : ",dirnames)
#     print("Files: ",filenames)
#     print("***************************************")

# # # This is useful for locating a file that you can’t remember where it was
# If you had a web app, and you wanted to keep track of the file info within a certain directory structure,
# then you could to thru the os.walk method and go thru all files and folders and collect file information.
#  # Access home directory location by grabbing home environment variable
# X = os.environ.get("HOME") # Returns a home directory path of the user
# print(X)  # /home/narendrab




# # # # # To properly join two files together use os.path.join(). & will be inserted one slash
# # file_path = os.path.join(os.environ.get("HOME"), "test.txt")

# # # # the benefit of os.path.join, is it takes the guess work out of inserting a slash
# # print(file_path)  # /home/narendrab/test.txte as a tuple
## os.path.exists("/tmp/test.txt")# # returns a boolean
## os.path.isdir("/tmp/test.txt")# # returns False
## os.path.isfile("/tmp/test.txt")# # returns True
## os.path.splitext("/tmp/test.txt")# # Splits file route of the path and the extension
# # returns ("/tmp/test", ".txt")# # This is alot easier than parsing out the extension. 
Splitting off and taking the first value is much better.



# # Very useful for file manipulation

# os.path has other useful methods
## os.path.basename# # This will grab filename of any path we are working on
## os.path.dirname("/tmp/test.txt")# # returns the directory /tmp
## os.path.split("/tmp/test.txt")# # returns both the directory and the fil



"""We can find out, From the root path(provided by user) 
what are the files (.txt/.py/.doc)available in each sub-folders.  """
import os

# path = "/home/narendrab/Python"
# for root,subdir,files in os.walk(path):
# # we can find-out .py files available in each subdirectory from provided root directory
##  for file in files:
#     if file.endswith(".txt"):
#        print("FileName ==> {}".format(os.path.join(root,file)))

## # same like above we can find-out .py files available in each subdirectory from provided root directory
##  for file in files:
#     if file.endswith(".py"):
#        print("FileName --> %s" %(os.path.join(root,file)))

def get_files_based_(path, extension):
    for root, subdir, files in os.walk(path):
        for file in files:
            if file.endswith(extension):
                print("FileName %s" % (os.path.join(root, file)))

# get_files_based_(path, ".txt")

                                Request: Please provide your valuable comments

Linux basic commands and concepts

                            List of basic Linux commands

                                                                 ==========================================

Linux is a multitasking and multiprocessing system (It allows multiple processes to operate simultaneously without interfering with each other)

                                           Directory-related Linux commands (Directory is nothing but a folder)

pwd    ==> Displays present working directory path
mkdir ==> The mkdir command will allow you to create directories. 
            Example: mkdir  Narendra ==> This command will create a directory called "Narendra".
rmdir: The rmdir command will delete an empty directory. 

below 2 commands are the same

rm -r <folder_name > To delete a directory and all of its contents recursively

rm -rf  <folder name>  To delete a directory and all of its contents recursively without requiring confirmation

cd ==>change directory  (To traverse from one directory to inside directory)

The cd command will allow you to change directories.

When you open a terminal you will be in your home directory.

To move around the file system you will use cd. 

    Examples:

• To navigate into the root directory, use cd / ↵
• To navigate to your home directory, use  cd  ↵ or cd ~ ↵
• To navigate up/back one directory level, use cd .. ↵
• To navigate to the previous directory (or before), use "cd - ↵
• To navigate through multiple levels of a directory at once, specify the full directory path that you want to go to.   For example, use, "cd home/Narendra/test_scripts/" to go directly to the test_scripts subdirectory.
•  As another example, "cd ~/Desktop" will move you to the Desktop subdirectory inside your home directory.

                                 Files related Linux commands 

     touch file_name.txt                                   ==> create a single file in pwd (present working directory)
     touch file_name1.txt  file_name_2.txt     ==> create multiple files in pwd 
     cat file_name1.txt                                      ==> display the content of the file

     cat file_name1.txt  file_name_2.txt         ==> display the content of the multiple files at a time

     cat file_name1.txt  file_name_2.txt  > filename_3.txt   ==> joins 2 files and stores the output in 3rd file.

     cp file_name1.txt  file_name_2.txt          ==> copy entire content/data from file_name1.txt  to  file_name_2.txt

     cp -r  directory_1 directory_2   

     mv file_name.txt narendra.txt             ==> to rename the file ( rename file_name.txt as narendra.txt)

     head file_name.txt                                 ==> show the first 10 lines of the file

     tail file_name.txt                                    ==> show the last 10 lines of the file

     wc file_name1.txt                                   ==> print the number of words, no of lines, file_size(number of bytes)

   

 find . -name "narendra.txt" ==> Find a file (narendra.txt /any extension-related files)  in current and sub-directories

 find /home -name *.jpg ==> Look for all .jpg files(any extension-related files) in the /home and directories below it.

ls            ==> Displays available files & folders available in pwd 
ls -l        ==>  Displays available files & folders available in pwd with permissions, sizes, date & time

ls -a       ==>  Displays available & Hidden files & folders available in pwd 

ll            ==>  Displays available & Hidden files & folders available in pwd with permissions, sizes, date & time

ls -lart   ==>  Displays all the files in the current directory but displays latest files at the below along with time stamp. 

ls -R      ==>  Displays all the files in the sub-directories along with the path. 

diff       ==> Compare the contents of two files line by line

         Example: diff  <file_name_1_with_extension>  <file_2_name_with_extension>

tar     ==> archive multiple files into a common Linux file format.

                    How to convert  a folder into a tar file

                tar  -czvf  new_name_or_existing_dir_name.tar.gz  directory_name 

               Ex: tar  -czvf  abc.tar.gz  narendra

                       tar  -czvf  narendra.tar.gz  narendra

                       ls

                To untar any tar file -->  tar -xvzf  tar_file_name         

               

How to unzip a zip file 

    sudo apt-get install unzip

    unzip file_name.zip ( or ) unzip file.zip -d destination_folder

 rm    ==> This command is to remove or delete a file in your directory.

            Ex: rm <file_name_with_extension>

readlink -f file_name  ==> It displays the complete file path 
Small clarification: pwd will display the file's path, but readlink -f file_name will display the path name and the file name at a time.

è What is chmod 777?

        chmod 777 means granting all permissions to read, write, and execute any file/directory by any user of the system

        sudo chmod 777 -R  Dir_name  (Note:  -R  means recursive    Change files and directories recursively)

General Linux commands

lscpu   ==> Displays the complete CPU-related information (Model name: XXX  XXXX XXXX 64-Core Processor)

who     ==>  At present parallelly  how many people using your machine (& IP address of them )

cat /etc/os-release  ==> See the complete OS-related information

    command output:     NAME="SLES/ UBUNTU/"

                                       VERSION="15-xxx"

                                       VERSION_ID="15.xx"

                                        PRETTY_NAME="xxxx Linux Enterprise Server 15xx

uname -r  ==> Displays the current OS kernel version (system information)==> 5.x.xx-xx-default

sudo dkms status    ==>  Displays which GPU and kernel version ==> amdgpu, 5.xx.x.xx.xx-1xxxxx  

man  command shows the manual instructions of Linux commands  Example: man ps

ps command provides information about the currently running processes, including their process identification numbers (PIDs).  

kill command is used to terminate the process manually Example:   kill -9 PID 

hostname  command shows the name of your host/network

ping  check connectivity to a server  ping <IP_address of any server>  Ex: ping 192.168.X.X

last  command shows last logins  (Note: displays last logins along with date& time)

id command shows details of the user.

top command monitor system resource usage

ifconfig command shows the IP addresses of all network interfaces. ( inet ip_address is essential to access)

                                        

                                        Upgrade UBUNTU OS

   cat /etc/os-release ==> It will display current OS details on the machine

   sudo apt-get update

   sudo apt-get upgrade -y

   Note1: For upgrade to dot version  sudo apt-get dist-upgrade -y  && sudo reboot 

       Ex: Ubuntu 20.04.4 LTS" to Ubuntu 20.04.5 LTS" 

   Note2: For  upgrade to new release version   sudo do-release-upgrade -y  &&  sudo reboot 

       Ex: Ubuntu 20.04" to Ubuntu 22.04" 

   cat /etc/os-release ==>  You have to check OS was upgraded or not?

 

Upgrade RHEL OS   

   cat  /etc/os-release ==> It will display current OS details on the machine

   sudo yum update

   sudo yum  upgrade -y

   Note1: For Update dot version  sudo yum  dist-upgrade -y  && sudo reboot 

   Note2: For  Upgrades to new release sudo do-release-upgrade -y  &&  sudo reboot 

   cat  /etc/os-release ==>  You have to check OS was upgraded or not?

                          Install Java / jdk (Java development kit) through CLI 

   In Ubuntu OS    --> sudo apt install openjdk-17-jdk

   In RHEL OS     --> sudo yum install java-17-openjdk-devel

                               or

  In RHEL OS     --> sudo rpm -ivh jdk-17_linux-aarch64_bin.rpm

                                       How to upgrade only kernel  directly 

 sudo apt install linux-image-<kernel_version> linux-headers-<kernel_version> linux-modules-extra-<kernel_version>

 Ex: sudo apt install linux-image-6.2.0-26-generic linux-headers-6.2.0-26-generic linux-modules-extra-6.2.0-26-generic

then reboot the machine sudo reboot then check  by using the command uname -r 

                                    Disk Usage-related Linux commands

    df  command gets a report on the system's disk space usage (it will display in KB (Kilobyte))

    df -h command gets a report on the system's disk space usage (it will display in GB (gigabyte))

    du command checks the disk space usage of a file or directory 

            Ex: du narendra.txt or du folder_name

    du -ah shows disk usage for all the files & directory

    du -sh shows disk usage of the current directory

How to connect remote machine/server from Linux Terminal 
ssh username@<IP address> press Enter, enter the password of the remote machine
ssh narendra@10.xx.xx.xx <Enter>

How to copy a file from a local to a remote machine
scp file_name.zip root@10.xx.41.xxx:/home/

How to copy a file within the same machine?
cp<space> file_name_to_be_copied<space>path_of _the_directory (where to copy)

How to copy a directory within the same machine?
cp<space> directory_name_to_be_copied<space> -r <space>path_of _the_directory (where to copy)

watch command :
==============
Syntax: watch [OPTIONS] COMMAND

 watch the command will temporarily clear all of the terminal content and run the provided command at regular intervals. When used without any option watch will run the specified command every two seconds.
Example: 1
In Terminal==> watch date <Enter> by default, every 2 seconds date will update. 

==> watch<space> -n <space>5 <space>date (for 5 seconds date will update)
==> watch <space>-n <space>5 <space>-t <space> date
Note: Ctrl+c to exit from it.

Example: 2 
watch python log_parser.py 
log_parser python will execute continuously.

hostnamectl  command will give the below information 

 Static hostname: 

       Icon name: computer-desktop

         Chassis: desktop/ server

      Machine ID: 

         Boot ID: 

Operating System: Ubuntu 22.04.3 LTS

          Kernel: Linux 6.5.0-xx-generic

    Architecture: x86-64

 Hardware Vendor: ASUS

  Hardware Model: System Product Name

sudo dmidecode -t chassis command will give the below information

# dmidecode 3.3

Getting SMBIOS data from sysfs.

SMBIOS 3.5.0 present.

Handle 0x0003, DMI type 3, 22 bytes

Chassis Information

        Manufacturer: Supermicro

        Type: Desktop/server

        Lock: Not Present

        Version: 

        Serial Number: 

        Asset Tag: Chassis Asset Tag

        Boot-up State: Safe

        Power Supply State: Safe

        Thermal State: Safe

        Security Status: None

        OEM Information: 

        Height: Unspecified

        Number Of Power Cords: 1

        Contained Elements: 0

        SKU Number: To be filled by O.E.M.

                                           Linux Keyboard Shortcuts

ctrl+alt+t        ==> To open a terminal in Linux-related OS 

ctrl+ Alt +t     ==> To open a new terminal window

ctrl+ Shift +t  ==> To open the terminal tab

ctrl + a        ==> Move the cursor to the beginning of the line

Ctrl + e        ==> Move the cursor to the end of the line

 ctrl + c       ==> To stop/come out / break the currently executing command (In the Terminal)

(if you have typed a long/short command in the terminal, but you don't want to run/ erase it, then you can use Ctrl+C in the Terminal)

ctrl +d         ==> to close the current terminal (or) exit   ==> logout of the current session.

ctrl +l          ==> will clear the terminal screen (or) clear command will clear the terminal screen

ctrl +r         ==> Recall the last command that matches the provided characters (handy command)

TAB           ==> autofill typing (while typing directory names/file names /commands type starting 2 letters then type TAB )

 !!                 ==> Repeat the last command

ctrl + k        ==> cut part of the line after the cursor & add it to the clipboard

ctrl + u       ==> cut part of the line before the cursor & add it to the clipboard

ctrl + w       ==> cut one word before the cursor & add it to the clipboard

ctrl + y       ==> paste from clipboard

 ctrl + o        ==> Run the previously recalled command (for some Linux flavors only)

Screen tool

sudo apt-get install screen  (it will take a couple of seconds to install)

to create a screen with a specific name then use the below command. 

command screen -S <screen_name>       (Capital S)

Example:    screen -S  narendra  

then run a script or run a command, which will take a longer duration.

while running the script you can come out of the screen, to create one more screen to perform another operation.

to come out of the 1st screen click on Ctrl+A  leave the control  then just press D to come out of the "Narendra screen"  

(Note: we called it [detached from 5150. Narendra])  but still, the command/script will be under process.

screen -S  Surendra  run the command  to create one more screen for executing scripts which may take a longer duration 

Click on Ctrl+AD to come out of the "Surendra screen"  

(note: we called it [detached from 5150. Surendra])  but still, the command will be under process.

if you want to see the list of background running screens then type the command: 

screen -ls  

There are screens on:

        5657.Narendra   (14/07/22 11:00:47 AM IST)      (Detached)

        5150.Surendra     (14/07/22 10:55:53 AM IST)      (Detached)

==>  To go to a particular screen &  to see what is happening on it type the command 

screen -r Narendra   (we called -r as re-attach)

==> To close any screen completely, go to the screen and type Ctrl +d  / type exit command. 

==>  To detach the attached screen, screen -d Narendra this command will detach the Narendra screen.

Python Related commands
====================

python -V(capital V) ==> Displays python 2.7.X version installed on your machine
python3 -V(capital V) ==> Displays python 3.X.X version installed on your machine.

 What is a log file?
 A log file is a computer-generated data file that contains information about usage patterns,
 activities, and operations within an operating system, application, server or another device.
 Log files show whether resources are performing properly and optimally.


 What is the purpose of a log file?
 Log files are used to record events or activities that occur within a computer system/ application/ program.
 Log files serve as a detailed record of what has happened within a system, 

 which can be used for troubleshooting problems or investigating security incidents.

Example1: let’s assume server crash happened! 

                   then "Log files are used to investigate what happened before server crashed"

 Note: In Linux based OS, all log files will be saved at "/var/log/" path
 Example2: After system turn ON, if you have faced any problem, 

                     then we need to check in boot.log (available at "/var/log/" )

                Request: Please provide your valuable comments