Details

Published: Monday, 27 September 2021 16:30

Hits: 826

sed:

streamline editor for noninteractive purpose. This is a linux pgm that was used in older days to manipulate files. Now a days, perl and other scripts can do the job just as easily. sed allows us to do same editing as from within vi, but allows us to do it  from cmd line by providing editing cmd and name of file. It's very easy to use, and should be your first choice whenever you want to search, replace for patterns in your file. Other scripting languages require quite a bit of code to many any manipulations to a file (i.e you have to open a file, copy lines one by one to other file, after modifying lines of interest, and then close all the files. Most scripting languages don't allow editing the same file content, while sed allows you to do that). So, 10 lines of code in some other language may require just 1 line of code in sed. sed has very easy syntax (you just need to regular expression), and very few cmds to get 99% of work done.

A very good reference is here: https://www.grymoire.com/Unix/Sed.html

You can also include sed cmds in any other script, as though they were native unix cmds (as sed is installed by default on all Linux distro). So, you can think of sed as ls, cp, etc which does file editing for you.

NOTE: There are 2 variants of sed: BSD sed and GNU sed. BSD sed is the original one, but it has lot of idiosyncrasies. You should always use the gnu sed version, and that's what's installed on most Linux distro. Most of the cmds you find online are assuming GNU sed (also called as gsed). Type "sed --version" on terminal to check sed version.

Terminal prompt$ sed --version
sed (GNU sed) 4.2.2
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.

syntax:

sed <options> 'command' filename => command tells sed what to do with lines in file, change it, remove it, etc. There are many optional options that can be provided which have to be preceeded by -. The command itself may be put in single quotes, or in double quotes (see under variable substitution to see when should double quotes be used).

ex: sed -i '/_slash/d' ~/tmp.txt => Here -i is the option, '...' is the cmd, and we are using this cmd on file tmp.txt

options:

Sed cmds are provided on terminal and it prints modified o/p on screen. If we want the modified o/p to be saved in a file, we can redirect sed o/p to another file (by usng linux > operator). There are various options to change the default behaviour.

• -i => If we want to modify the original file,  we have to use -i (inplace) option. NOTE: -i is not for "case insensitive". sed doesn't have an option to be called with case insensitive option (BSD sed may have a -I option for case insensitivity applied globally, but it doesn't work on gnu sed).
• -r => sed by default uses BRE (Basic RE, see in Regular Expression section), but using -r makes it use ERE (ERE is recommended). 
• -e => To specify multiple sed cmds

command:

command follows the usual syntax: i.e 'flag1/<original_pattern>/<modified_pattern>/flag2' => This is similar to pattern replacement syntax that you see in Perl, Python, etc. "flags" in beginning or end specify what we want to do with the pattern. d=delete (used in flag2), s=substitute (used in flag1), etc. More details later.

deleting patterns: We use '/<pattern>/d' to delete matched pattern

sed -i -r '/_slash/d' ~/tmp.txt => delete _slash from tmp.txt (-i means same file tmp.txt is modified. -r means use extended regular expression (ERE)).

delete blank lines: sed '/^$/d' in.txt > out.txt => all lines in file in.txt that start and end with nothing in b/w are deleted, and o/p is passed on to new file out.txt

substituting pattern: We use 's/<orig_pattern>/<new_pattern>/' to substituting matched pattern

sed -r "s/'/ /g" in.txt > out.txt =>Here we are replacing ' with space. So, we had to use double quotes, since using single quotes gave an error => unmatched '. g is used to say do it globally.

sed -e 's/.*/PRE: & SUF \\/g' in.txt > out.txt =>Here in every line, we are replacing he start and end of line with "PRE:" and "SUF \" respectively. We have to use double backslash, since single backslash is itself an escape char. If we don't use \\, and instead use \, then / following \ is treated as literal, so we get "unterminated cmd" error.

sed -r 's/\$\{.*slash\}//g'   ~/tmp${num}.txt > out.txt => here we globally replace ${.....slash} with nothing, i.e remove that pattern. Here instead of doing inline replacement, we pass on the results to another file named out.txt

Remembering matching patterns: This is done by enclosing the pattern to be rembered using () and then recalling it back using \1, \2 etc where \1 is the 1st pattern within (), \2 is the second pattern and so on.

sed -r 's/(.*)\s+(.*)/\2 -SPACE \1/g' ~/orig.txt > ~/mod.txt => Here we are replacing all lines which have patterns of form " aa/a?aa  bbb/ccc" with "bbb/ccc -SPACE aa/a?aa". NOTE: we didn't use \(.*\) since we are using ERE (by using -r) where "(", ")" is recognized as special char. If we were using BRE, then we needed to use \(.*\)

Substituting with a variable: So far, we used single quotes in cmd section. We said that single and double quotes don't matter in the cmd section of sed. We use one over the other depending on whether the cmd itself contains single or double quotes. i.e  double quotes is used if you need to use a single quote in command itself. One other place double quotes is used is when you need to substitute the value of a var in the sed cmd. It behaves just like other scripting languages, where one kind of quote allows "var substitution" while other kind doesn't.

ex: set a ="my lord"; sed -i "s/abc.*$/ME $a/" z.log => HereAnything with "abc" until the end of line is subtituted with char "ME my lord". If we used single quotes, then "ME $a" will get substituted.

Print or delete between specific Markers: This is very used cmd when we have markers in a file and we want to cut out that protion of file, and replace it with something else. This is mostly used when some automated section of a file is updated via scripts, and we want the rest of the file untouched.

ex: sed -e '/START_MARKER/,/END_MARKER/d' File1 => d flag says to delete lines (p flag is to print lines, but doesn't work - FIXME) b/w the 2 markers: START_MARKER and END_MARKER. If END_MARKER is some pattern which is not there in the file, then everything from START_MARKER to the EOF is deleted. Use some marker as ZZZZ which is not there in file, and then contents from START_MARKER to the EOF is deleted.

Substitute text between specific Markers with contents from another file: This is not easy, as I tried several 1 liners on internet and nothing worked. The only code that works is below and it ONLY works in Bash shell. It'll error out in csh as it needs 2 lines for sed cmd to work (which apparently breaks in csh):

Link: https://superuser.com/questions/440013/how-to-replace-part-of-a-text-file-between-markers-with-another-text-file

Here, original file is edited. It has contents b/w "START of Pattern' and "END of Pattern" that are replaced by contents of replacement_file. NOTE the 2nd line on sed cmd. You have to put that code in 2nd line (i.e after a newline), else it won't work !!

lead='START of Pattern'
tail='END of Pattern'
sed -i "/$lead/,/$tail/{ /$lead/{p; r replacement_file.txt
  }; /$tail/p; d }"  original_file.txt

Details

Published: Sunday, 26 September 2021 16:10

Hits: 870

JSON:

short for JavaScript Object Notation, and is usually pronounced like the name “Jason”. We want to learn JSON syntax because it's used very frequently, takes 5 minutes to learn, and most of the websites store and send data in JSON format. XML is another text format used widely in websites for storing and sending data. JSON is better as it's faster and easier to read. A good link for JSON basics is here:

https://www.w3schools.com/Js/js_json_intro.asp

JSON defines a syntax, which can be used to store data in a text file for any purpose. JSON’s format is derived from JavaScript object syntax, but it is entirely text-based. It is a key-value data format that is typically rendered in curly braces. You can write a utility or pgm to extract key-value pair from this json file. Most of the web programming languages, as well as scripting/interpreted languages (as python, perl, etc) offer inbuilt support for parsing JSON files.

JSON is not just for Javascript, and has nothing to do with Javascript, exept that it's syntax is derived from Javascript, and so we have included it here right after the Javascript section. It may be helpful to read into material on Javascript.

ex: test.json => Here we have 3 keys and 3 values corresponding to those 3 keys. Key "first_name" has a value "Sammy", and so on. keys and values are separated by colon (:). Note there is a comma separating key/value pairs, but no comma for the last entry. 

{

  "first_name"  :  "Sammy",
  "last_name"   :  "Shark",
  "online"      :  true
}

Many webservers return data as JSON strings, which are JSON object literal or JSON array enclosed within single quotes. The 2 types of JSON strings are as follows:

1. JSON object literal: The JSON object literal is enclosed in curly braces. example as shown above. This is in key-data format.
2. JSON array: The JSON array is enclosed within square brackets. ex: ["Ford", "BMW", "Fiat"]. This is not for key-data, It's used when you have a set of data, not in key:data format.
   

NOTE: JSON strings can also be a combination of the 2 types above by having object literal nesten within array or vice versa, i.e ["Ford", {"make": "model1"}, "BMW, {"make":"model2"}]. Depending on whether that element is "object literal" or "array", we access it differently. Shown later.

To make these JSON strings, we would enclose them in single quotes as in ex here => ' { first_name": "sammy" ... } '

In JS, these json strings are parsed using JSON.parse() function which converts them into JS objects.

Even though JSON is derived from JS, there are few differences:

1. In JS or HTML files, the json object is set as a var. i.e in above ex, we can write person = { first_name: "Sammy", ... }. Then the variable person holds this js object. We access this object various ways as:
   
   • Dot notation: person.first_name or assign value as person.first_name="meme"
   • Array notation: person["first_name"] or assign value as person["first_name"] = "meme"
2. In JS, Strings always need to be in quotes( either single quotes or double quotes). But in JSON, strings always need to be in double quotes. Reason:??
3. In JS, keys can be strings, numbers or identifier names. while in JSON keys can only be strings. So, in jS keys do not need to be in quotes (except for when the keys are strings. When they are strings, we can use either single quotes or double quotes). But in JSON files, keys can only be strings and as such always need to be in double quotes.
4. In JSON, values may be string, number, another json object (can have more nested key value pair), array, boolean (true or false) or null. Values are in double quotes when they are also strings.
   1. Value as JSON object literal => { "employee":{"name":"John", "age":30, "city":"New York"} }
   2. Value as array => { "employees":["John", "Anna", "Peter"] } => This is in cases where you have more than 1 value. In this case, we don't have a way to represent these using "object literal" as they can only be in key:data format. So, we added this array format, so that all kinds of data can be represented using combo of { } and [ ].
      

Accessing JSON entries:

A very important concept in JSON is to know how to access various entries in this JSON object literal. When the JSON object has arrays and literals, with other JSON objects nested within each other. it becomes complicated. There's online JSON parsers and validators, which will show JSON files neatly formatted, so that you can access various values.

Here's one such validator: https://jsononline.net/json-validator

ex: var person =

{
  "data": {
     "current_condition": [{
        "cloudcover": "0",
         ....
         },{
        "cloudcover": "5", ...
        }]
       "past-condition": [{....}]
   }
   "info" { .... }
}

These elements can be accessed via array notation as shown:

person["data"]["current_condition"][1]["cloudover"] => This returns "5". Here 1st element "data" is "object literal" with key data format, so we access it in associative array style, i.e person["data"]. This returns the whole "data" content in "data". "data" itself is object literal in key:data format, as it's within { }. "current-condition" is the key, while data is an array, as it's within [ ]. So, person["data"]["current_condition"] points to "data" for key "current_condition". The data is an array as shown.

[{"cloudcover": "0",...},{"cloudcover": "5", ...}]

This array has 2 entries, so we refer it using array[0] and array[1]. person["data"]["current_condition"][1]points to 2nd index of array 
and is equal {"cloudcover": "5", ...}. Now this is an "object literal" with multiple entries in key:data format. We just provide the key of the data 
we are interested in. Here we do:

person["data"]["current_condition"][1]["cloudcover"] which returns data "5" for key "cloudcover"

Details

Published: Thursday, 23 September 2021 18:04

Hits: 1124

XML Extensible Markup Language

This is a specifies that says how you to arrange the data so that it's easeir for tools to read the data. The file is still a text file, but it's hareder for humans to read these files.

Here's a good link explaining the basics:

https://www.datacamp.com/community/tutorials/python-xml-elementtree

NOTE: If you run the examples in link above, do note that xml file has closing </collection> missing. If you don't add that, you will get an error when parsing the xml file.

XML:

XML looks very much like HTML files. XML creates a tree-like structure that is easy to interpret and supports a hierarchy

XML files are called XML documents

Details

Published: Monday, 20 September 2021 13:42

Hits: 664

OpenSuse:

This is the 1 of the 2 OS that has support of major big company called SuSE. CentOS is supported by RedHat while OpenSuse is supported by SuSe.

However,it has been frustrating for me to get OpenSuse to work. Basic video do not work on Firefox. You will have to download bunch of other stuff to make it work. Also, installation gave me issues. There is very little support available online, and takes a lot of your time resolving minor issues. I wasn't able to get skype working on it. Considering all these, I've abandoned this as there's no unique advantage that OpenSuse provides, which isn't provided by others. I've provided instructions below for completeness. Follow instructions on OpenSuse website to dowload leap 15:

http://www.opensuse-guide.org/installation.php

Follow the instructions on link above. However, based on my experience, I would advise to stay away from it, since there are so many other alternatives available.