Update README.md

README.md

@@ -157,4 +157,31 @@ print('\n\n', tokenizer.decode(output_tokens[0], skip_special_tokens=True))
 This output is like the training data. If you run without applying the Lora, it will usually look worse. If you retrain the lora, know that your new lora is not going to output the same results, despite you using the same settings.
 Inference should usually be deterministic when using the same lora, or using without lora.
 
-    
+
+
+
+Also, If you want to download and use the loras from a visible folder, here's the inference script:
+
+```
+import torch
+from peft import PeftModel, PeftConfig
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+peft_model_id = "./loramodel"
+config = PeftConfig.from_pretrained(peft_model_id)
+model = AutoModelForCausalLM.from_pretrained(config.base_model_name_or_path, return_dict=True, load_in_8bit=True, device_map='auto')
+tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
+
+# Load the Lora model
+model = PeftModel.from_pretrained(model, peft_model_id)
+
+
+batch = tokenizer("Two things are infinite: ", return_tensors='pt')
+
+with torch.cuda.amp.autocast():
+  output_tokens = model.generate(**batch, max_new_tokens=50)
+
+print('\n\n', tokenizer.decode(output_tokens[0], skip_special_tokens=True))
+```
+
+add your adapter_config.json and your adapter_model.bin to a folder in your current directory named `loramodel`, or whatever you choose.