Kubernetes || Remove Namespace stuck in terminating state after delete

Currently, we are getting some issues while deleting the namespace in Kubernetes(AKS). When we run "kubectl delete ns" command, it got stuck and after sometime, if we abort this it will remain in terminating state forever.

Below is the kubernetes version we are using-

$ kubectl  version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Here, we will see how we can use the workaround to remove the stuck namespace.
1. Create test Namespace-

$ kubectl create ns test
namespace/test created
$ kubectl get ns test
NAME   STATUS   AGE
test   Active   45s

2, Try to delete namespace. You will notice that below command got stuck there. I will hit "ctrl+c" to abort and then check status.

$ kubectl delete ns test
namespace "test" deleted
^C
$ kubectl get ns test
NAME   STATUS        AGE
test   Terminating   3m

You saw that namespace is still there in terminating state. Now in below steps we will discuss about the workaround.

1. Get the json output for above namespace-

$ kubectl get ns test -ojson > test.json
$ cat test.json
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2019-01-25T11:34:16Z",
        "deletionTimestamp": "2019-01-25T11:36:27Z",
        "name": "test",
        "resourceVersion": "6575638",
        "selfLink": "/api/v1/namespaces/test",
        "uid": "25e3d87e-2095-11e9-bbb7-36dd4f9e3a95"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

2. You saw the "finalizers" object in above json file. Please remove the line with "kubernetes". So, the file will look like below -

$ sed -i  "/"kubernetes"/d" test.json && cat test.json
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2019-01-25T11:34:16Z",
        "deletionTimestamp": "2019-01-25T11:36:27Z",
        "name": "test",
        "resourceVersion": "6575638",
        "selfLink": "/api/v1/namespaces/test",
        "uid": "25e3d87e-2095-11e9-bbb7-36dd4f9e3a95"
    },
    "spec": {
        "finalizers": [
        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

3. Start the kube proxy using comand "kubectl proxy"

$ kubectl proxy
Starting to serve on 127.0.0.1:8001

4. Now on other terminal run below command-

$ curl -H "Content-Type: application/json" -X PUT --data-binary @test.json http://127.0.0.1:8001/api/v1/namespaces/test/finalize
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "test",
    "selfLink": "/api/v1/namespaces/test/finalize",
    "uid": "25e3d87e-2095-11e9-bbb7-36dd4f9e3a95",
    "resourceVersion": "6577114",
    "creationTimestamp": "2019-01-25T11:34:16Z",
    "deletionTimestamp": "2019-01-25T11:36:27Z"
  },
  "spec": {

  },
  "status": {
    "phase": "Terminating"
  }

5. Check is namespace has been deleted or not-

$ kubectl get ns test
Error from server (NotFound): namespaces "test" not found

Here you go!! Its deleted. This is some kind of bug in K8S. I haven't checked if this have been already fixed on higher version.

TMUX (part-2) Installation and Basics

In the last Article (tmux-Terminal Multiplexer Intro - Part-1), we just discussed what is tmux, comparison with terminals and how this can be helpful in our day to day activities.

Today, lets discuss about the Installation and some basics stuff about the tmux.

Installation-

So, if you are using mac then install tmux as below:

              $ brew install tmux 

If using ubuntu then you can install using "apt-get" as below:

              $ sudo apt-get install -y tmux 

To verify, if the installation is completed successfully by checking the version-

              $ tmux -V

              tmux 2.6

If you want then you can also do the installation using the source code.

Starting tmux:

Start tmux just typing using command "tmux":
              $ tmux 
So when you will just hit enter, you'll notice that you will be get landed to another window as show in below screen shot. Yes, you are right, so that will be your workspace where you are going to work. You can do everything here, as you do with your normal terminals.

Fig-1 tmux terminal(See at the bottom the red highlighted)

To exit this, just type "exit" command and then you will be back to your standard terminal.

Another thing to notice, if you doing stuff for just a short period, then maybe this is not the best way to work with tmux sessions. Instead, We can create "named sessions", that we can then use later on.

Create Named Sessions:

On a single terminal, we can create multiple sessions by giving them a particular name and then organize them as needed. Now, using below command, I'll create 2 sessions with name "Dev" and "Test". 

              $ tmux new-session -s Dev

              $ tmux new-session -s Test

For now it may be doesn't seem useful. But what we you want to execute some long running process e.g. any monitoring process or any long DB query and don't want to scare the screen all the time. In these kind of situation, you can take help on tmux feature which helps us to run jobs in background and help us to catch the things latter on. Let's see how we can do this.

Detaching and Attaching the Sessions:

Now, lets see how we can detach and then later attach to the same screen. Now, I have two sessions "Dev" and "Test". I am going to run "top" command in "Dev" and monitor the partition in another "Test" Session.

Now, we can detach from both sessions using the combination of PREFIX(Ctrl+b) followed by "d". In below screenshot, I have detached both sessions and now back to the standard terminal.

On the standard terminal, you list down the active sessions as well.

 Reattaching to Existing Sessions:

Now, you know to detach from the session and run the stuff in the background. On the next day lets support if want to see whats going on inside these sessions, then you have flexibility yo reattach to these as shown below:

              $ tmux attach -t Dev

              $ tmux attach -t Test

That's all for now. Please try your self and check if this can be useful for you.

tmux-Terminal Multiplexer Intro - Part-1

tmux - Everything in one terminal( Mouse-Free Development ;) )

What is tmux- 

                       tmux is a terminal multiplexer. It helps us to use one single terminal/window to launch multiple terminals. e.g. in your first terminal launch tmux and then load top command. After that, you can create another window and load some database CLI console. Now we have two programs running in the same original terminal. An interesting thing is that within the same session, you can switch back and forth these programs.

Nowadays each modern Operating System provides terminals with multitab functionality. its not something new with tmux. But main feature that tmux provides is running more than one programs simultaneous. In tmux, we have a feature called "pane". "Pane" is again a new window inside tmux window. We can have multiple panes which can be arranged in horizontal or vertical. This again helps us to run multiple programs on same screen. And yes, everything without mouse :).

Below is the example of pane. One single tmux window having 6 pane-

Another interesting thing about the tmux is that you can detach from the current session. This will help us to keep the long process running in the background. This will be a bit familiar to the guys who have used "GNU-screen" already. tmux is a bit similar to GNU-screen, but it provides many more features without any complex configuration. I think which makes it really cool and fun to work with. Along with this tmux has a client-server model. So everything can be controlled from one central location.

In the next article, we will discuss the Installation and basic usages of tmux to start with. So subscribe and keep visiting.